General Bayesian Loss Function Selection and the use of Improper Models

Abstract Statisticians often face the choice between using probability models or a paradigm defined by minimising loss function. Both approaches are useful and, if can be re-cast into proper model, there many tools to decide which model is more appropriate for observed data, in sense of explaining data's nature. However, when leads an improper no principled ways guide this choice. We address task combining Hyvärinen score, naturally targets infinitesimal relative probabilities, and general Bayesian updating, provides unifying framework inference on losses models. Specifically we propose ℋ-score, selection criterion prove that it consistently selects (possibly improper) closest data-generating truth Fisher's divergence. also associated ℋ-posterior learns optimal hyper-parameters featuring functions, including challenging tempering parameter generalised inference. As salient examples, consider robust regression non-parametric density estimation where popular functions define data hence cannot dealt with standard tools. These examples illustrate advantages robustness-efficiency trade-offs enable kernel estimation, opening new avenue non-parametrics.